1 link tagged with all of: iceberg + s3 files + data storage
Click any tag below to further narrow down your results
Links
Andy Warfield recounts how genomic researchers struggled with moving large datasets between local filesystems and S3, spurring the creation of S3 Files to present S3 buckets as POSIX-compatible file stores. The post then covers how Amazon added S3 Tables for managed Iceberg tables and built-in vector indexing to cut storage friction and speed up analytics and search.
- S3 Files was inspired by genomics researchers at UBC who wasted hours copying sequencing data between NFS and S3 because their GATK4/Spark pipeline still required POSIX filesystem semantics despite massive serverless parallelism.
- Agent-driven development is lowering the barrier for domain experts to build apps, making it more critical to decouple persistent data from any single application or workflow.
- S3 handles over 25 million Parquet requests daily, often via Hive tables, but open formats like Iceberg still force customers to manage object-level APIs, compaction jobs, and metadata services themselves.
- S3 Files aims to provide a filesystem-style interface on top of S3's scale and durability, eliminating manual copy scripts and unifying table-format intelligence under one system.